The present invention is directed to a fault handling monitor for use in a high availability, networked file server and, in particular, to a fault handling monitor transparently using multiple technologies for fault handling in a file server organized as multiple hierarchical and peer domains wherein each domain includes domain centered fault handling mechanisms operating cooperatively across domains.
A continuing problem in computer systems is in providing secure, fault tolerant resources, such as communications and data storage resources, such that communications between the computer system and clients or users of the computer system are maintained in the event of failure and such that data is not lost and can be recovered or reconstructed without loss in the event of a failure. This problem is particularly severe in networked systems wherein a shared resource, such as a system data storage facility, is typically comprised of one or more system resources, such as file servers, shared among a number of clients and accessed through the system network. A failure in a shared resource, such as in the data storage functions of a file server or in communications between clients of the file server and the client file systems supported by the file server, can result in failure of the entire system. This problem is particularly severe in that the volume of data and communications and the number of data transactions supported by a shared resource such as a file server are significantly greater than within a single client system, resulting in significantly increased complexity in the resource, in the data transactions and in the client/server communications. This increased complexity results in increased probability of failure and increased difficulty in recovering from failures. In addition, the problem is multidimensional in that a failure may occur in any of a number of resource components or related functions, such as in a disk drive, in a control processor, or in the network communications. Also, it is desirable that the shared resource communications and services continue to be available despite failures in one or more components, and that the operations of the resource be preserved and restored for both operations and transactions that have been completed and for operations and transactions that are being executed when a failure occurs.
Considering networked file server systems as a typical example of a shared system resource of the prior art, the filer server systems of the prior art have adopted a number of methods for achieving fault tolerance in client/server communications and in the file transaction functions of the file server, and for data recovery or reconstruction. These methods are typically based upon redundancy, that is, the provision of duplicate system elements and the replacement of a failed element with a duplicate element or the creation of duplicate copies of information to be used in reconstructing lost information.
For example, many systems of the prior art incorporate industry standard RAID technology for the preservation and recovery of data and file transactions, wherein RAID technology is a family of methods for distributing redundant data and error correction information across a redundant array of disk drives. A failed disk drive may be replaced by a redundant drive, and the data in the failed disk may be reconstructed from the redundant data and error correction information. Other systems of the prior art employ multiple, duplicate parallel communications paths or multiple, duplicate parallel processing units, with appropriate switching to switch communications or file transactions from a failed communications path or file processor to an equivalent, parallel path or processor., to enhance the reliability and availability of client/file server communications and client/client file system communications. These methods, however, are costly in system resources, requiring the duplication of essential communication paths and processing paths, and the inclusion of complex administrative and synchronization mechanisms to manage the replacement of failed elements by functioning elements. Also, and while these methods allow services and functions to be continued in the event of failures, and RAID methods, for example, allow the recovery or reconstruction of completed data transactions, that is, transactions that have been committed to stable storage on disk, these methods do not support the reconstruction or recovery of transactions lost due to failures during execution of the transactions.
As a consequence, yet other methods of the prior art utilize information redundancy to allow the recovery and reconstruction of transactions lost due to failures occurring during execution of the transactions. These methods include caching, transaction logging and mirroring wherein caching is the temporary storage of data in memory in the data flow path to and from the stable storage until the data transaction is committed to stable storage by transfer of the data into stable storage, that is, a disk drive, or read from stable storage and transferred to a recipient. Transaction logging, or journaling, temporarily stores information describing a data transaction, that is, the requested file server operation, until the data transaction is committed to stable storage, that is, completed in the file server, and allows lost data transactions to be re-constructed or re-executed from the stored information. Mirroring, in turn, is often used in conjunction with caching or transaction logging and is essentially the storing of a copy of the contents of a cache or transaction log in, for example, the memory or stable storage space of a separate processor as the cache or transaction log entries are generated in the file processor.
Caching, transaction logging and mirroring, however, are often unsatisfactory because they are often costly in system resources and require complex administrative and synchronization operations and mechanisms to manage the caching, transaction logging and mirroring functions and subsequent transaction recovery operations, and significantly increase the file server latency, that is, the time required to complete a file transaction. It must also be noted that caching and transaction logging are vulnerable to failures in the processors in which the caching and logging mechanisms reside and that while mirroring is a solution to the problem of loss of the cache or transaction log contents, mirroring otherwise suffers from the same disadvantages as caching or transaction logging. These problems are compounded in that caching and, in particular, transaction logging and mirroring, require the storing of significant volumes of information while transaction logging and the re-construction or re-execution of logged file transactions requires the implementation and execution of complex algorithms to analyze, replay and roll back the transaction log to re-construct the file transactions. These problems are compounded still further in that these methods are typically implemented at the lower levels of file server functionality, where each data transaction is executed as a large number of detailed, complex file system operations. As a consequence, the volume of information to be extracted and stored and the number and complexity of operations required to extract and store the data or data transactions and to recover and reconstruct the data or data transactions operations is significantly increased.
Again, these methods are costly in system resources and require complex administrative and synchronization mechanisms to manage the methods and, because of the cost in system resources, the degree of redundancy that can be provided by these methods is limited, so that the systems often cannot deal with multiple sources of failure. For example, a system may provide duplicate parallel processor units or communications paths for certain functions, but the occurrence of failures in both processor units or communications paths will result in total loss of the system. In addition, these methods of the prior art for ensuring communications and data preservation and recovery typically operate in isolation from one another, and in separate levels or sub-systems. For this reason, the methods generally do not operate cooperatively or in combination, may operate in conflict with one another, and cannot deal with multiple failures or combinations of failures or failures requiring a combination of methods to overcome. Some systems of the prior art attempt to solve this problem, but this typically requires the use of a central, master coordination mechanism or sub-system and related complex administrative and synchronization mechanisms to achieve cooperative operation and to avoid conflict between the fault handling mechanisms, which is again costly in system resources and is in itself a source of failures.
The present invention provides a solution to these and other related problems of the prior art.
The present invention is directed to a fault handling monitor transparently using multiple technologies for fault handling in a shared system resource, such as a file server, residing in a networked system and providing services to a plurality of clients communicating with the system resource through a network, and to the method for transparently using multiple technologies for fault handling in a shared system resource.
According to the present invention, the system resource is organized as a cluster of multiple hierarchical and peer domains wherein each domain includes domain centered fault handling mechanisms operating cooperatively across domains. The system resource includes a network domain including a plurality of client/resource communications paths and supporting client/resource communications between the system resource and a client of the system resource, a resource service domain performing low level resource services operations, and a control/processing domain supporting the client/resource communications of the network domain, performing high level resource service operations and providing communications for resource service operations between the network domain and the resource service domain. The control/processing domain includes peer domains connected to the plurality of communications paths of the network domain and performing related operations in mutual support of the frictions of the network domain, including supporting the client/resource communications operations of the network domain and providing communications between the peer domains through an inter-domain communications link. Each peer domain includes a monitoring mechanism for detecting a communications failure in a peer domain and directing communications affected by the communications failure through one or more of a plurality of alternate paths, the alternate paths including an alternate one of the plurality of client/resource communications paths and the inter-domain communications link and the peer domain in which the monitoring mechanism resides.
In a presently preferred embodiment, the shared system resource is a file server and includes a network domain supporting client/server communications between the file server and a client of the file server, a storage domain supporting the file transaction operations of the control/processing domain and supporting client file systems, and a control/processing domain supporting the client/server communications of the network domain and performing high level file transaction operations and providing communications for file transaction operations between the network domain and the storage domain. The control/processing domain includes a pair of peer processing blade domains performing operations in support of the client/server communications functions of the network domain and higher and lower level file transaction operations. Each peer processing blade domain includes a higher level domain supporting the client/server operations of the network domain and performing high level file transaction operations, and a lower level domain performing lower level file transaction operations and supporting communications inter-domain communications between the peer processing blade domains, wherein the higher level and lower level domains of the peer processing blade domains operate in mutual support in providing communications for file transaction operations between the network domain and the storage domain. Each higher level domain of each peer processing blade domain includes a monitoring mechanism for detecting a communications failure in the other peer domain and directing communications affected by the communications failure through one or more of a plurality of alternate paths to the other peer domain, the alternate paths including an alternate one of the plurality of client/resource communications paths and the inter-domain communications link and the peer domain in which the monitoring mechanism resides. The alternate communications paths for a failure affecting the client/server communications include an alternate one of the plurality of client/resource communications paths and the inter-domain communications link and the peer domain in which the monitoring mechanism resides, and the alternate communications paths for a failure affecting the inter-domain communications between the peer processing blade domains include an alternate one of the plurality of client/resource communications paths.